fix(cuda.core): pickle Device by UUID by Andy-Jost · Pull Request #3198 · NVIDIA/cuda-python

Andy-Jost · 2026-10-22T23:40:54Z

Summary

A pickled Device named its device by ordinal, so a process with a different CUDA_VISIBLE_DEVICES unpickled it as a different GPU. Device.__reduce__ now sends the UUID, and the receiver maps it back with _device_from_uuid() from #3126. Unpickling raises RuntimeError, naming the UUID and the visible devices, when the device is not visible.

Changes

_device.pyx: Device.__reduce__ returns (_device_from_uuid, (uuid,)). The class docstring describes the pickled form.
1.3.0 release-note entry.
Tests: an in-process round trip returns the same thread-local singleton; a child with reversed visibility receives the device with the parent's UUID; a child that cannot see the device gets the error.

Related Work

Closes #3196. Built on #3126 (merged 2026-10-23), which added _device_from_uuid().

🤖 Generated with Claude Code

A DeviceMemoryResource sent through multiprocessing, and a pickled VirtualMemoryResource, named their device by ordinal, and the VirtualMemoryResource also sent its peers as ordinals. Ordinals are per process, so a receiver with a different CUDA_VISIBLE_DEVICES rebuilt the resource on the wrong GPU or granted peer access to the wrong GPU. Send the owner and the peers by UUID and map them back in the receiver with a new private helper. Raise a clear RuntimeError when no visible device has the UUID. The tests reverse the device order in the child through CUDA_VISIBLE_DEVICES and check the imported buffer's device UUID. Closes NVIDIA#3097 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…ce-uuid

The check-cython-not-none hook from main requires it on a def parameter annotated with a cdef class. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…ce-uuid Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A generator in `peers` was consumed by the first allocation. Later allocations granted no peer access, and a pickled resource sent no peer UUIDs, so the receiver mapped the memory without peer access. `__post_init__` now converts `peers` to a tuple, which also covers options given as a dict, through `replace()`, or per call to `modify_allocation()`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Device.__reduce__ sent the ordinal, so a process with a different CUDA_VISIBLE_DEVICES unpickled a different GPU. It now sends the UUID, and the receiver maps it back with _device_from_uuid(), which raises RuntimeError when the device is not visible. Closes NVIDIA#3196. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…uuid Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

coderabbitai · 2026-10-22T23:48:41Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Source mode: Auto
Repository: NVIDIA/cuda-python
Branch: main
File: .coderabbit.yaml
Commit: d5f37e0
Layers: Repository YAML
Review profile: CHILL
Plan: Enterprise
Run ID: d12262ef-5335-44e8-8c74-5ef50695eb1b

📥 Commits

Reviewing files that changed from the base of the PR and between 670b7d9 and 2197a4d.

📒 Files selected for processing: 4

cuda_core/cuda/core/_device.pyi
cuda_core/cuda/core/_device.pyx
cuda_core/docs/source/release/1.3.0-notes.rst
cuda_core/tests/memory_ipc/test_device_uuid.py

🚧 Files skipped from review as they are similar to previous changes: 3

cuda_core/cuda/core/_device.pyi
cuda_core/docs/source/release/1.3.0-notes.rst
cuda_core/cuda/core/_device.pyx

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

📝 Summary

Summary by CodeRabbit

Bug Fixes
- Pickled devices now retain their identity across processes on the same machine by UUID, even when device ordinals differ. Unpickling resolves the UUID to a visible device and raises an error if that device is unavailable.
- Pickling and unpickling a device within the same process and thread returns the identical device object.

Device pickling now identifies devices by UUID. Unpickling resolves the UUID to a visible device, regardless of its ordinal, and raises RuntimeError if no matching device is visible. Tests cover same-process and cross-process behavior.

Changes: Device UUID pickling

Layer / File(s)	Summary
UUID serialization and reconstruction `cuda_core/cuda/core/_device.pyx`, `cuda_core/cuda/core/_device.pyi`, `cuda_core/tests/test_device.py`	`Device.__reduce__` stores the device UUID. Documentation describes UUID resolution and the error for an invisible device. A same-process test checks that unpickling returns the identical device object.
Cross-process transfer validation `cuda_core/tests/memory_ipc/test_device_uuid.py`, `cuda_core/docs/source/release/1.3.0-notes.rst`	IPC tests check UUID preservation across reordered device ordinals and failure when the device is not visible. Cleanup for the buffer tests now runs in `finally` blocks. Release notes describe the pickling change.

Assessment against linked issues

Objective	Addressed	Explanation
Pickle a `Device` by UUID, resolve it to the matching device in the receiving process, and raise `RuntimeError` if it is not visible [`#3196`]	✅

Possibly related issues

cuda.core: IPC pickling rebuilds memory resources with the sender's device indices #3097: The changes cover Device transfer in the shared-memory workflow when process-local device indices differ. The issue’s resource-index handling is separate.

Possibly related PRs

NVIDIA/cuda-python#3126: This PR uses the UUID-resolution helper and contract from #3126 to reconstruct pickled Device objects.

Suggested reviewers: rwgk

Priority: ➖ Normal

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 2197a

The UUID-based transfer behavior is covered for reordered and unavailable devices; no identified issue blocks merging.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

ℹ️ Review info

⚙️ Run configuration

Source mode: Auto
Repository: NVIDIA/cuda-python
Branch: main
File: .coderabbit.yaml
Commit: afa6249
Layers: Repository YAML
Review profile: CHILL
Plan: Enterprise
Run ID: 6c0f850a-0fcc-4f2e-abda-c0c16ee5a016

📥 Commits

Reviewing files that changed from the base of the PR and between afa6249 and ed9de37.

📒 Files selected for processing: 10

cuda_core/cuda/core/_device.pyi
cuda_core/cuda/core/_device.pyx
cuda_core/cuda/core/_memory/_device_memory_resource.pyi
cuda_core/cuda/core/_memory/_device_memory_resource.pyx
cuda_core/cuda/core/_memory/_virtual_memory_resource.pyi
cuda_core/cuda/core/_memory/_virtual_memory_resource.pyx
cuda_core/docs/source/release/1.3.0-notes.rst
cuda_core/tests/memory_ipc/test_device_uuid.py
cuda_core/tests/test_device.py
cuda_core/tests/test_memory.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

github-actions · 2026-10-22T23:59:32Z

Doc Preview CI
Preview removed because the pull request was closed or merged.

NVIDIA#3126 merged as d5f37e0 (squash). Keep the Device additions in test_device_uuid.py and the NVIDIA#3196 release-note entry. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

coderabbitai

🧹 Nitpick comments: 1

cuda_core/tests/memory_ipc/test_device_uuid.py: 1

143-143: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

suggestion: Assert the missing UUID without fixing the error wording.

If the error wording changes, this assertion fails even when the RuntimeError still names dev0.uuid. Assert that the message contains dev0.uuid instead. As per coding guidelines: “Avoid asserting on internal call counts, private helper invocation order, or error message substrings that are not part of the contract.”

Source: Coding guidelines

ℹ️ Review info

⚙️ Run configuration

Source mode: Auto
Repository: NVIDIA/cuda-python
Branch: main
File: .coderabbit.yaml
Commit: d5f37e0
Layers: Repository YAML
Review profile: CHILL
Plan: Enterprise
Run ID: ce6ee340-a832-47e2-9455-2202f7dd0ff7

📥 Commits

Reviewing files that changed from the base of the PR and between ed9de37 and 670b7d9.

📒 Files selected for processing: 2

cuda_core/docs/source/release/1.3.0-notes.rst
cuda_core/tests/memory_ipc/test_device_uuid.py

🚧 Files skipped from review as they are similar to previous changes: 1

cuda_core/docs/source/release/1.3.0-notes.rst

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

…resources on every path - The Device docstring and the release note say the pickled form is for processes on the same machine (asked on NVIDIA#3196). - The buffer tests in test_device_uuid.py release their buffers and pools in finally blocks, so a failed assertion cannot leave an owned pool open for the next test's leak check (CodeRabbit on NVIDIA#3198). - The module docstring cites NVIDIA#3196 for the Device requirement. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Removed preview folders for the following PRs: - PR #3198 - PR #3209

Andy-Jost and others added 7 commits October 15, 2026 14:54

Merge remote-tracking branch 'origin/main' into ajost/ipc-pickle-devi…

34a69a8

…ce-uuid

fix(cuda.core): mark the IPC reconstructor's handle parameter not None

160a21c

The check-cython-not-none hook from main requires it on a def parameter annotated with a cdef class. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into ajost/ipc-pickle-devi…

96d5136

…ce-uuid Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge branch 'ajost/ipc-pickle-device-uuid' into ajost/device-pickle-…

ed9de37

…uuid Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Andy-Jost added this to the cuda.core 1.3.0 milestone Oct 22, 2026

Andy-Jost added bug Something isn't working P1 Medium priority - Should do cuda.core Everything related to the cuda.core module labels Oct 22, 2026

Andy-Jost self-assigned this Oct 22, 2026

coderabbitai Bot reviewed Oct 22, 2026

View reviewed changes

Comment thread cuda_core/tests/memory_ipc/test_device_uuid.py

Merge remote-tracking branch 'origin/main' into ajost/device-pickle-uuid

670b7d9

NVIDIA#3126 merged as d5f37e0 (squash). Keep the Device additions in test_device_uuid.py and the NVIDIA#3196 release-note entry. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Andy-Jost mentioned this pull request Oct 23, 2026

cuda.core: a pickled Device names its device by ordinal #3196

Closed

coderabbitai Bot reviewed Oct 23, 2026

View reviewed changes

jpascucci-nv approved these changes Oct 23, 2026

View reviewed changes

Andy-Jost mentioned this pull request Oct 23, 2026

EPIC: VMM feature completeness #2980

Open

1 of 2 tasks

Andy-Jost enabled auto-merge (squash) October 23, 2026 20:04

Andy-Jost merged commit 0bdf9e9 into NVIDIA:main Oct 24, 2026
200 of 202 checks passed

github-actions Bot pushed a commit that referenced this pull request Oct 25, 2026

Clean up PR preview folders for 2 closed/merged PRs

7e4cfba

Removed preview folders for the following PRs: - PR #3198 - PR #3209

Navigation Menu

Repository navigation

fix(cuda.core): pickle Device by UUID - #3198

fix(cuda.core): pickle Device by UUID#3198
Andy-Jost merged 9 commits into
NVIDIA:mainfrom
Andy-Jost:ajost/device-pickle-uuid

Andy-Jost commented Oct 22, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 22, 2026 •

edited

Loading

Summary by CodeRabbit

Assessment against linked issues

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

github-actions Bot commented Oct 22, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Andy-Jost commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Related Work

Uh oh!

coderabbitai Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Assessment against linked issues

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Andy-Jost commented Oct 22, 2026 •

edited

Loading

coderabbitai Bot commented Oct 22, 2026 •

edited

Loading

github-actions Bot commented Oct 22, 2026 •

edited

Loading